Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change ordering LIFO to FIFO on delayed jobs #416

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mariocamaraneto
Copy link

@mariocamaraneto mariocamaraneto commented Apr 30, 2021

Push in left and pop from right

Without this we can have Live Lock because some Jobs added first never wil be attendend if new jobs won't stop arriving

Q A
Is bugfix? yes
New feature? no
Breaks BC? no
Tests pass? yes

Push in left and pop from right
@mariocamaraneto mariocamaraneto changed the title Change ordering LIFO to FIFO. Change ordering LIFO to FIFO on delayed worekr Apr 30, 2021
@mariocamaraneto mariocamaraneto changed the title Change ordering LIFO to FIFO on delayed worekr Change ordering LIFO to FIFO on delayed worker Apr 30, 2021
@mariocamaraneto mariocamaraneto changed the title Change ordering LIFO to FIFO on delayed worker Change ordering LIFO to FIFO on delayed jobs Apr 30, 2021
@rob006
Copy link

rob006 commented May 1, 2021

Without this we can have Live Lock because some Jobs added first never wil be attendend if new jobs won't stop arriving

This is not true. Expired/delayed job will be executed after execution of all new jobs that were pushed in the meantime. So while order of execution may not match order of push (it was never a guarantee in queue), this job will eventually be executed.

@mariocamaraneto
Copy link
Author

The delayed job added to the queue can change the order that they were pushed. Eventually the can be executed, but this may delay ther execution significantly.

In servers with a high amount of delayed jobs being added to the queue, the jobs can be delayed for hours. I saw this happening in our servers, where new jobs were being processed almost immedietly, while old jobs were queued for hours.

I attached a file with a simulation, where in the queue with delayed jobs we can see that job p3 can be stuck in waiting "forever" if new delayed jobs keeps being pushed. Like in "live lock".

In queues without delay, all new jobs are pushed to the left and reserved in right. This is exactly I coded in this PR.

Queue simulator with delay.pdf

@mariocamaraneto
Copy link
Author

I made a test to simulate this situation, but I don't like the solution because use many SLEEP and decrease the decreases the speed of tests.
You can see here: https://github.com/mariocamaraneto/yii2-queue/pull/1/files

@samdark samdark added this to the 2.3.1 milestone May 4, 2021
@samdark
Copy link
Member

samdark commented May 4, 2021

A test is necessary. Also please add a line for CHANGELOG.

@samdark samdark modified the milestones: 2.3.1, 2.3.2 May 5, 2021
@samdark samdark removed this from the 2.3.2 milestone Oct 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants